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1.  Executive  Summary 

The  intent  of  this  paper  is  to  identify  what  measures  are  necessary  to  aid  an  acquisition  agency  and  a 
contractor  when  it’ s  decided  that  a  program  should  follow  an  evolutionary  acquisition  strategy. 
Evolutionary  acquisition  is  a  strategy  that  develops  and  deploys  a  core  capability  with  the  intent  to  field 
additional  capahilities  as  stakeholder  needs,  expectations,  constraints,  and  interfaces  are  better 
understood. 

Accordingly,  we  have  developed  four  essential  measures  for  adopting  an  evolutionary  acquisition 
strategy: 

1 .  The  amount  of  risk  on  the  program 

The  intent  of  evolutionary  acquisition  is  to  reduce  the  risk  on  the  project.  Unless  this  is  measured, 
there  is  no  quantitative  way  to  know  whether  the  risk  has  been  reduced. 

2.  The  requirements  changes  by  stakeholder  type. 

Evolutionary  acquisition  assumes  that  there  is  significant  involvement  by  all  of  the  stakeholders  on 
the  program.  This  should  be  measured. 

3.  The  number  of  requirements  added,  deleted,  and  modified  per  block. 

As  the  system  evolves,  and  additional  blocks  are  delivered,  it  is  important  to  understand  how  the 
initial  requirements  of  the  system  have  been  modified  as  the  system  evolves. 

4.  The  discrepancy  reports  against  the  architecture. 

It  is  important  to  know  that  the  initial  architecture  will  support  the  total  evolution  of  the  system. 

We  looked  at  reasons  for  evolutionary  acquisition,  and  developed  questions  that  might  be  asked  by  the 
project  managers  of  a  program  that  has  decided  to  follow  a  evolutionary  acquisition  strategy  to  determine 
additional  measures  that  will  help  the  project  manager  of  an  evolutionary  acquisition  are  discussed  later 
in  this  paper. 

2.  Overview 

The  process  used  in  the  investigation  of  measures  identified  critical  success  factors  for  evolutionary 
acquisition.  These  factors  came  from  two  SEI-sponsored  Office  of  the  Secretary  of  Defense  (OSD) 
workshops,  and  from  interviews  about  evolutionary  acquisition.  These  factors  were  mapped  into  a 
measurement  framework  taken  from  Practical  Software  and  System  Measurement  (PSM)  [PSM  00] . 

Using  the  PSM  process,  measures  were  identified  to  provide  insight  into  these  factors  (Section  6)  by 

selecting  from  the  existing  measures 

•  modifying  the  existing  measures,  or 

•  defining  new  measures. 

In  several  cases,  new  measures  were  not  required,  proving  that  existing  measures  can  often  be  adapted  to 
aid  in  the  management  of  an  evolutionary  acquisition. 

Members  of  the  Lockheed  Martin  Undersea  Systems  were  interviewed  for  their  experience  in  managing 
evolutionary  acquisition  and  spiral  development  [Roper  00].  This  Manassas,  Virginia-based  organization 
is  rated  at  a  Capability  Maturity  Model®  for  Software  Level  5  and  possesses  a  strong  measurement 
program.  The  group’s  existing  measurement  program  covers  evolutionary  acquisition  and  development, 
as  well  as  other  acquisition  and  development  strategies.  The  issue  is  a  matter  of  deciding  to  change  the 
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emphasis  of  their  measurement  program  instead  of  developing  new  measures.  One  measure  that  they  use, 
which  is  not  currently  a  part  of  the  PSM  measures,  is  risk.  Risk  is  tracked  over  time  to  determine  a  trend. 
Their  approach  to  defining  and  managing  evolutionary  projects  is  discussed  later. 

This  paper  will  address  the  application  of  measures  that  are  currently  in  practice,  as  well  as  potential  new 
measures  that  will  aid  in  the  management  of  an  evolutionary  acquisition. 

3.  Evolutionary  Acquisition  Definition 

Evolutionary  acquisition  is  a  strategy  that  develops  and  deploys  a  core  capahility  with  the  intent  to  field 
additional  capahilities  as  stakeholder  needs,  expectations,  constraints,  and  interfaces  are  better 
understood.  Future  capahilities  are  deployed  in  steps  called  “blocks”  as  shown  in  Figure  1  [Ferguson  00]. 
Future  requirements  to  existing  system  capabilities  come  from  end-user  experience,  emerging  new 
technologies  (e.g.  Science  and  Technology  activities),  and  support  for  new  operational  capabilities. 

“Evolutionary  acquisition  is  an  approach  that  fields  an  operationally  useful  and 
supportable  capability  in  as  short  a  time  as  possible.  This  approach  is  particularly  useful 
if  software  is  a  key  component  of  the  system,  and  the  software  is  required  for  the  system 
to  achieve  its  intended  mission.  Evolutionary  acquisition  delivers  an  initial  capability 
with  the  explicit  intent  of  delivering  improved  or  updated  capability  in  the  future...  Block 
1  provides  the  initial  deployment  capability  (a  usable  increment  of  capability  called  for  in 
the  Operational  Requirements  Document).”  [DOD5000.2  00] 

One  of  the  basic  benefits  of  evolutionary  acquisition  is  that  a  user’ s  hands-on  experience  with  the 
system  generates  potential  new  capabilities  and  requirements.  It  is  interesting  to  note  that  due  to 
the  block  overlap  shown  in  Figure  1 ,  feedback  and  perhaps  new  requirements  learned  from  using 
the  deployed  system  will  not  be  implemented  until  two  blocks  (at  the  earliest)  after  the 
identification.  This  could  be  remedied  if  there  was  time  between  the  deployment  of  one  block  and 
the  planning  of  the  next  block,  if  the  appropriate  contract  mechanisms  were  in  place. 
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Figure  1:  Evolutionary  Block  Increment  Approach 


An  important  requirement  for  specifying  measures  is  to  understand  the  criteria  and  issues  for  establishing 
an  acquisition  as  evolutionary.  The  issues,  upon  which  recommendations  for  measurement  are  based, 
cannot  be  identified  until  this  acquisition  approach  is  understood.  A  definition  was  developed  in  a  SEI- 
sponsored  OSD  workshop  held  in  September  2000.  The  workshop  attempted  to  identify  attributes  of 
evolutionary  acquisition. 

A  discussion  group  at  the  workshop  focused  on  the  definition  of  evolutionary  acquisition.  The  table 
below  summarizes  their  findings  by  comparing  an  evolutionary  to  an  incremental  acquisition  strategy. 
[Place  00] 
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Evolutionary  Acquisition 

Incremental  Acquisition 

Definition:  Evolutionary  acquisition  is  an 
acquisition  approach  that  deploys  a  core  capability 
and  incrementally  inserts  additional  capabilities,  as 
requirements  are  refined. 

Definition:  Incremental  acquisition  is  an 
acquisition  approach  that  deploys  a  full  capability 
that  is  incrementally  fielded  based  upon  firm 
requirements  for  each  block. 

Attribute  1:  Multiple  deployments  accommodating 
evolving  requirements. 

This  is  driven  by  opportunistic  technology 
insertion,  resolution  of  the  many  unknowns 
that  exist  with  large  complex  systems, 
evolving  threats,  and  the  user's  understanding 
of  delivered  capabilities. 

There  is  variability  in  the  degree  of  flexibility 
and  time  phasing  of  the  deployments. 

Attribute  1:  Requirements  for  multiple 
deployments  are  strictly  defined. 

This  is  driven  by  few  unknowns,  a  defined 
threat,  planned  technology  insertion,  and  the 
user’s  understanding  of  the  final  capabilities. 

The  variability  is  in  the  time  phasing  of  the 
increments. 

Attribute  2:  Acquisition  is  risk  driven. 

This  captures  the  intent  to  define  blocks  based 
on  risk  reduction  and  having  risks  define  the 
content  of  blocks. 

There  is  variability  in  the  number  of  blocks. 

Attribute  2:  Acquisition  is  requirements  driven. 

This  captures  the  concept  that  capabilities  for 
each  increment  are  defined. 

The  variability  is  the  number  of  increments. 

Attribute  3:  Stakeholders  are  involved  in  the 
decision  point  at  the  end  of  each  block. 

The  decisions  are  the  satisfaction  of 
completion  criteria  for  an  block  and  the 
decision  to  “field”  or  “not  field”  the  developed 
system. 

The  variability  is  who  the  stakeholders  are  and 
the  decision  criteria  (risk,  funding,  capability, 
etc.) 

Attribute  3:  Stakeholders  are  involved  in  the 
decision  point  at  the  end  of  each  increment. 

The  decision  is  whether  to  “field”  or  “not 
field”  the  developed  system.  The  completion 
criteria  are  established  with  the  pre-allocation 
of  requirements. 

The  variability  is  who  the  stakeholders  are  and 
the  decision  criteria  (risk,  funding,  satisfaction 
of  requirements,  etc.) 
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Evolutionary  Acquisition 

Incremental  Acquisition 

Attribute  4:  Emphasis  is  on  total  life  cycle 
activities. 

Attribute  4:  Emphasis  is  on  total  life  cycle 
activities. 

All  life  cycle  activities  are  planned  for  each 
block  and  the  planning  occurs  for  each  block 
throughout  system  development. 

All  increments  are  preplanned  and  the  life 
cycle  activities  must  be  planned  for  each 
increment. 

The  variability  is  in  the  amount  of  change  to 
operational  procedures,  training,  testing, 
maintenance  support,  re-certification,  human 
interface,  etc. 

The  variability  is  in  the  change  to  testing, 
support,  training,  etc. 

3.1  The  Role  of  High  Level  Requirements 

One  approach  to  looking  at  role  of  requirements  in  evolutionary  acquisition  requirements  is  to  look  at  the 
Operations  Requirements  Document  (ORD).  [Ferguson  00]  The  ORD  defines  overall  requirements  at 
Milestone  0.  These  requirements  described  the  vision  and  ultimate  objectives  of  the  system.  It  includes  a 
full  definition  of  full  capability,  as  well  as  a  firm  definition  of  requirements  to  be  satisfied  by  each  block, 
including  the  Initial  Operational  Capability  for  each  block.  The  acquisition  strategy  shall  define  each 
block  of  capability  and  how  it  will  be  funded,  developed,  tested,  produced,  and  operationally  supported. 
[DOD5000.2  00,  Sec  2.2. 1.2.3] 

The  detailed  requirements  become  known  over  time  based  on  the  threat,  strategy,  available  capability,  and 
available  technology.  [Ferguson  00] 

ORDs  contain  eight  or  fewer  key-performance  parameters  (KPP)  [Ferguson  00].  KPPs  are  characterized 
by  describing  required  capabilities,  warfighting  capabilities,  and  achievable  and  realistic  success  criteria 
(explainable  by  analysts).  ORDs  need  to  specify  threshold  values  (minimum  acceptable  value)  and 
objective  values  (desired  value)  for  use  in  design  and  acceptance  criteria.  One  required  KPP  will  always 
be  “Interoperability.”  [DOD5000.2  00,  Sec  2.1.1] 

Other  important  attributes  of  the  evolutionary  acquisition  strategy  are  that  it  may  span  many  years,  meet 
unforeseen  threats,  and  must  have  a  robust  architecture  capable  of  adapting  to  new  emerging 
technologies.  If  the  ORD  cannot  be  met,  either  because  of  changes  in  the  environment  (such  as  threat 
changes)  or  because  of  significant  changes  in  the  technology,  the  program  must  be  able  to  be  terminated 
based  on  impracticality  without  consequences  to  careers  or  developer  reputation. 

The  Lockheed  Martin  organization  in  Manassas,  Virginia  [Roper  00]  has  developed  a  slightly  different 
model  of  evolutionary  development.  In  their  model,  a  set  of  System  requirements  is  developed.  These 
system  requirements  are  allocated  to  a  set  of  deliveries  (blocks).  A  block  is  defined  to  be  a  deliverable 
that  can  be  used  in  the  field.  At  the  beginning  of  each  block,  the  block’ s  system  requirements  are 
allocated  to  software,  hardware,  and  refined  to  the  level  where  the  design  of  this  block  can  be 
accomplished.  They  have  defined  a  new  review  that  takes  place  at  the  end  of  each  block.  At  this  review, 
the  stakeholders  accept  the  current  block,  and  the  detailed  requirements  for  the  next  block  are  reviewed 
and  agreed  to.  This  version  of  evolutionary  development  has  some  distinct  advantages 

•  Since  the  total  system  requirements  are  known,  the  initial  architecture  is  designed  knowing  the  full  set 
of  system  requirements,  and  will  have  a  high  probability  of  not  having  significant  modifications  for 
the  life  of  the  program. 
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•  As  each  block  goes  through  the  design  phase,  the  requirement  of  the  future  blocks  can  be  used  as 
constraints  on  the  current  design. 

•  The  other  life  cycle  components  will  be  more  stable,  because  the  overall  system  requirements  are  well 
defined. 


4.  DoD  Policy  on  Evolutionary  Acquisition 

The  DoD  is  advocating  a  new  acquisition  approach  in  the  updated  5,000  series  of  policies.  This  new 
approach  has  the  goals  of  reducing  cost  and  cycle  time  while  delivering  improved  performance.  Seven 
key  focus  areas  were  identified  that  would  contribute  to  these  objectives  implement  time-phased 
requirements  and  evolutionary  acquisition 

•  strengthen  the  focus  on  modular,  open-systems  design 

•  strengthen  implementation  of  supporting  tools 

•  integrate  test  and  evaluation 

•  enhance  management  of  interoperability  and  system-of-systems  issues 

•  integrate  acquisition  and  logistics 

•  further  streamline  the  acquisition  process 

The  proposed  new  acquisition  strategy,  evolutionary  acquisition,  is  shown  in  the  Figure  2  below.  In  the 
Science  and  Technology  (S&T)  phase,  new  ideas  and  technology  are  started  and  matured.  The 
Demonstration  and  Risk  Reduction  phase  looks  at  alternatives,  prototyping  solutions,  maturing 
technology,  and  evolving  requirements.  At  the  end  of  this  phase,  a  commitment  to  an  acquisition  program 
is  made  based  on  initial  user  requirements,  usefulness,  mature  technology  currently  available,  and 
affordability  of  the  system.  An  evolutionary  acquisition  program  is  then  commenced  that  is  committed  to 
funding  the  first  delivery  at  a  reduced  cost  compared  to  the  total  program,  with  a  shorter  cycle  time,  and 
that  produces  a  usable  increment  of  capability. 

Figure  2:  Acquisition  Life  Cycle 
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This  paper  recommends  an  initial  set  of  measures  for  use  during  the  evolutionary  acquisition  phase  of 
procurement.  These  measures  are  based  on  the  issues  a  program  executive  officer  will  face  in  an 
evolutionary  acquisition  environment.  This  paper  proceeds  by  first  attempting  to  define  what  establishes 
an  acquisition  strategy  as  evolutionary.  With  this  foundation,  the  critical  success  factors  for  evolutionary 
acquisition  are  identified.  The  issues  and  questions  a  program  manager  might  have  concerning  these 
success  factors  are  discussed.  Measurement  categories  and  measures  are  identified  that  support  insight 
into  the  program  manager’s  concerns.  Finally,  the  rationale  is  provided  for  how  each  recommended 
measure  supports  insight  into  the  evolutionary  acquisition  approach. 

The  evolutionary  acquisition  strategy  implies  evolutionary  development  of  requirements,  i.e.  all  of  the 
detailed  implementation  requirements  are  not  known.  Insisting  on  specifying  all  of  the  detailed 


6 


Evolutionary  Acquisition  Measures 


requirements  before  acquisition  commences  leads  to  the  traditional  acquisition  strategies,  e.g.  waterfall  or 
incremental. 

5.  Evolutionary  Acquisition  Specific  Issues 

Several  critical  success  factors  were  identified  in  the  February  [Hansen  00]  and  September  [Workshop 
00]  of  2000  workshops  sponsored  by  the  Center  for  Software  Engineering  and  the  SEE  They  are  risk 
management,  a  trusting  culture,  involved  stakeholders,  technology  readiness,  flexible  requirements, 
schedule,  system  breakage,  and  impact  to  life  cycle  support  activities.  Eor  the  purposes  of  defining 
measures,  these  critical  success  factors  are  the  specific  evolutionary  acquisition  issues  that  will  be  used  to 
determine  the  candidate  measures. 

5.1  Risk  Management 

Evolutionary  and  spiral  approaches  to  acquisition  and  development  are  intended  to  mitigate  risk.  Risk 
management  should  be  used  as  an  essential  element  for  managing  evolutionary  acquisition.  The 
ability  to  measure  and  track  risk  evolution  over  time  is  critical  to  the  success  of  the  program. 

Understanding  that  difficult  technical  risks  can  break  system  and  software  architectures,  the  highest 
risks  should  be  identified  and  resolved  first. 

This  implies  the  need  for  an  architecture  that  is  well  understood  and  accommodates  risk  mitigation 
plans.  As  the  program  progresses,  there  should  be  a  steadily  decreasing  exposure  to  risk  and  the 
architecture  should  remain  stable.  Some  of  this  type  of  risk  can  be  measured  in  the  risk  section,  others 
of  this  type  of  risk  would  be  measured  in  the  system  breakage  section. 

Possible  questions 

•  What  are  the  high  exposure  risks  and  how  many  are  there? 

•  How  is  risk  exposure  decreasing? 

•  Are  risks  migrating  from  block  to  block? 

5.2  Trusting  Culture  and  Involved  Stakeholders 

The  acquirer,  user,  supplier,  and  maintainer  are  a  team.  A  trusting  culture  cooperates  in  different  areas 
of  responsibility,  embraces  new  ideas,  and  welcomes  outsiders.  In  a  trusting  culture,  technical  risks 
can  be  identified  without  fear  of  career  or  contractual  consequences.  A  risk  or  problem  is  addressed 
with  constructive  management  action  and  not  penalization.  The  “kill  the  messenger”  syndrome  must 
be  avoided.  The  culture  should  foster  a  collaborative  work  environment  where  commitments  are 
respected  and  new  ideas  or  negative  data  are  used  to  effectively  manage  the  program. 

The  key  to  successful  system  development  is  collaboration  between  the  DoD  and  the  contractors. 
Anecdotal  evidence  suggests  that  when  collaboration  has  occurred,  systems  have  been  developed  in 
an  economical  and  timely  fashion. 

Stakeholders  should  be  involved  in  the  establishment  of  the  overall  system  requirements,  and  in  the 
decision  of  which  requirements  will  be  implemented  with  every  block.  Stakeholders  are  defined  as 
those  groups  that  have  a  genuine  interest  in  the  successful  deployment  of  a  system,  have  a  shared 
understanding  of  what  the  system  will  do,  and  agree  to  measures  of  success.  Stakeholders  come  from 
groups  representing  technical  or  support  activities,  suppliers,  customers,  or  end  users. 

In  addition  to  the  establishment  of  system  requirements,  evolutionary  acquisition  requires  agreed- 
upon  commitments  by  stakeholder  groups  to 

•  participate  in  the  identification  and  resolution  or  issues  and  actions 

•  participate  in  block  demonstrations  and  testing 

•  upgrade  or  modify  life  cycle  factors  that  are  affected 


7 


Evolutionary  Acquisition  Measures 


Possible  questions 

•  What  are  effective  principles  and  practices  of  integrated  teams? 

•  Are  there  open  communications  among  stakeholders? 

•  Who  are  the  relevant  stakeholders? 

•  What  is  the  distribution  of  stakeholder  organizations? 

•  Are  different  stakeholder  responsibilities  being  met? 

•  What  will  be  the  stakeholder  turnover  within  and  between  blocks? 

5.3  Flexible  Requirements 

The  evolutionary  acquisition  model  must  start  with  a  set  of  known  system-level  requirements.  The 
specific  requirements  for  each  block  are  developed  as  a  result  of  the  current  risks  on  the  project,  the 
stakeholders  understanding  of  the  existing  version  of  the  system,  and  the  readiness  of  the  enabling 
technology.  It  is  important  to  track  requirements  moving  from  block  to  block.  This  shifting  of 
requirements  to  later  blocks  will  indicate  that  there  will  be  a  cost  increase  and  overall  program 
slippage. 

The  deployment  of  a  block  creates  a  new  baseline  of  requirements,  (i.e.  those  that  have  been 
implemented.)  New  requirements  will  develop  as  blocks  are  deployed.  As  the  system  is  used,  the 
baseline  requirements  will  be  modified.  There  is  a  need  to  measure  how  the  requirements  change  as 
the  deployment  of  blocks  progresses. 

Possible  questions 

•  How  many  requirements  will  be  implemented  in  a  block? 

•  How  are  requirements  traced  across  blocks  with  the  possibility  of  requirements  being  modified  or 
deleted  in  later  blocks? 

•  Will  evolving  requirements  impact  regression  testing? 

5.4  System  Breakage 

Breakage  occurs  when  the  latest  block  is  deployed  and  the  end-user  says,  “What  happened  to  the  old 
functionality?”  It  is  important  to  understand  at  deployment  time  how  the  latest  block  of  a  product 
differs  from  earlier  versions.  Measures  of  system  breakage  would  show  the  impact  of  the  latest  block 
on  the  products  produced  in  previous  blocks. 

Possible  questions 

•  Which  block  introduced  the  problem? 

•  Is  the  problem  due  to  evolution  of  a  requirement  or  new  functionality? 

•  Is  the  existing  architecture  able  to  support  the  proposed  new  functionality  and  technology? 

5.5  Technology  Readiness 

Acquisition  of  product  blocks  has  to  use  mature  technology.  It  needs  to  be  demonstrable  that  the 
hardware  is  manufacturable,  and  neither  the  hardware  nor  software  will  need  extraordinary,  open- 
ended  efforts  during  the  development  and  manufacturing  process  phases.  Experience  has  shown  that 
projects  are  likely  to  fail  unless  the  underlying  technology  has  reached  Level  6  on  a  scale  of  nine 
levels  of  “technology  readiness”,  as  described  below  [Hansen  00,  Sec.  2.3.4]. 
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Level 

Level  Description 

1 

Basic  principles  observed  and  reported 

2 

Technology  concept  and/or  application  formulated. 

3 

Analytical  and  experimental  critical  function  and/or  characteristic  proof  of 
concept. 

4 

Component  and/or  breadboard  validation  in  laboratory  environment. 

5 

Component  and/or  breadboard  validation  in  relevant  environment. 

6 

System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment. 

7 

System  prototype  demonstration  in  an  operational  environment. 

8 

Actual  system  completed  and  "mission"  through  test  and  demonstration. 

9 

Actual  system  “mission”  through  successful  mission  operations. 

Possible  questions 

•  Will  appropriate  test  beds  and  laboratories  be  available  when  needed? 

•  What  is  the  “readiness”  rating  for  each  technology  component  being  considered  for  tbe  next 
block? 


5.6  Schedule 

One  of  the  primary  drivers  for  evolutionary  acquisition  is  to  allow  increased  functionality  to  be  placed 
in  the  field  as  rapidly  as  possible.  As  a  part  of  the  acquisition  planning,  the  duration  between  blocks, 
objectives  of  each  block,  and  the  life  cycle  support  for  each  block  need  to  be  defined.  Since  the 
detailed  requirements  for  each  block  are  not  know  ahead  of  time,  the  block  duration,  block  objectives, 
and  changes  to  life  cycle  support  factors  need  to  be  agreed  upon  before  the  block  development 
commences.  The  testing  process  is  refined  to  include  both  regression  testing  of  previous  blocks,  and 
tbe  testing  of  new  capabilities. 

Possible  questions 

•  How  is  tbe  overall  schedule  for  the  project  characterized? 

•  What  is  the  duration  of  the  block? 

•  How  is  test  progress  measured  with  a  continuously  growing  system? 

•  Will  the  number  of  requirements  be  known  for  each  block  so  that  the  number  of  implemented 
requirements  can  be  tracked? 

•  What  are  the  success  criteria  for  each  block? 

5.7  Impact  to  Life  Cycle  Factors 

Deciding  on  the  functionality  of  the  next  block  affects  the  other  supporting  life  cycle  resources.  The 
actual  impact  on  operational  support  (measured  in  terms  of  effort  required  or  number  of  life  cycle 
components  that  are  changed)  should  be  fully  considered..  Operational  support  includes  areas  such  as 
maintenance,  supply,  transportation,  sustaining  engineering,  configuration  and  data  management, 
manpower,  training,  environmental,  health,  safety,  disposal,  and  security  factors.’ 

Possible  questions 

•  Wbat  is  the  cost  and  schedule  allocated  to  life  cycle  factors  in  a  block? 

•  When  is  the  cost  and  schedule  for  the  new  block  estimated? 


’  DOD  Directive  5000.2,  “2.3.2  Evolutionary  Sustainment,”  Final  Draft. 
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•  How  are  life  cycle  factors  impacted  for  each  block  planned? 

6.  Selecting  Measures 

6.1  Mapping  to  PSM  ICM  structure 

Specific  measures  for  individual  programs  would  be  selected  using  the  current  PSM  process  of  mapping 
specific  program  issues  and  risks  to  the  common  issue  areas,  measurement  categories,  and  specific 
measures.  With  some  exceptions,  the  measures  for  an  evolutionary  acquisition  are  not  different  than  for 
other  types  of  acquisitions.  What  is  different  are  the  attributes  collected  for  that  measure.  There  are  some 
measures  that  would  be  applicable  to  evolutionary  acquisition  that  are  not  in  the  current  PSM  structure. 
These  new  issues  and  categories  are  potential  extensions  to  the  existing  PSM  Issue -Category-Measure 
(ICM)  structure. 

Table  6.1  shows  the  PSM  ICM  structure  with  the  recommended  and  candidate  set  of  measures  for  use  in 
providing  program  management  insight  into  evolutionary  acquisition.  Some  of  the  evolutionary 
acquisition  issues  mapped  to  multiple  PSM  Common  Issues.  In  most  cases  the  mapping  was  to  existing 
PSM  Common  Issues  and  Categories.  The  measures  are  annotated  to  show  their  status  with  current  PSM 
measures: 

•  *  =  New  issue  area,  measurement  category,  or  measure 

•  T  =  Tailoring  of  attributes  of  an  existing  PSM  measure 

•  (Blank)  =  Existing  PSM  measures 
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Table  6.1 

Specific 

Issues 

Common 
Issue  Area 

Measurement 

Category 

Recommended 

Measure 

Other  Candidate 

Measures 

Risk 

Management 

Risk  (*) 

Risk  (*) 

Risk  exposure  (*) 

Number  of  risks  /block  (*) 

Number  of  risks  identified  against  a  critical  technology 
component  (*) 

Risk  Carryover  from  previous  block  (*) 

Flexibile 

Requirements 

Product  Size 
and  Stability 

Functional 

Size  and 
Stability 

Number  of  Requirements 
Added,  Deleted  & 
Modified  per  block. 

Requirements  allocated  by  block. 

Ratio  of  requirements  satisfied  in  a  block  to  total  system 
requirements  (T) 

Number  of  allocated  block  requirements  deferred  to  later 
blocks  (T) 

Number  of  baseline  requirements  changed. 

Number  of  system  requirements. 

Requirements  changes  by  stakeholder  type  (T) 

Trusting 

Culture  and 

Involved 

Stakeholders 

Product  Size 
and  Stability 

Collaborative 
Culture  (*) 

Requirement  Changes  by 
Stakeholder  Type  (T) 

Documented  commitments  by  relevant  stakeholders  (*) 
Documentation  and  resolution  of  issues,  and  action  items  by 
stakeholder  (*) 

Participation  by  stakeholders  in  demonstrations  or  testing  (*) 
Alignment  of  project  performance  with  projected 
stakeholder  needs,  objectives,  and  requirements  (*) 

Technology 

Readiness 

Resources 
and  Cost 

Enviroment 

Availability 

N/A 

Availability  of  appropriate  test  beds  (T) 

11 


Evolutionary  Acquisition  Measures 


Table  6.1 

Specific 

Issues 

Common 
Issue  Area 

Measurement 

Category 

Recommended 

Measure 

Other  Candidate 

Measures 

Flexible 

Requirements 

Schedule  and 
Progress 

Milestone 

Performance 

N/A 

Inter-block  milestones  and  changes  in  block  schedule  (T) 
Block  deployment  milestones 

Incremental 

Capability 

N/A 

Build  Component  -  function 

Work  Unit 
Progress 

N/A 

Test  case  status 

System 

breakage 

Product 

Quality 

Defects 

Discrepancy  Reports 
Against  Architecture  (T) 

Percentage  of  unchanged  test  cases  (from  previous  block) 

(T) 

Discrepancy  reports  against  previously  delivered 
functionality  (T) 

Tracking  the  amount  of  rework  from  one  block  to  the  next 
(T) 

Impact  to  Life 
Cycle  Factors 

Product 

Quality 

Life  Cycle 
Impacts  (*) 

N/A 

Impact  of  newly  allocated  requirements  to  a  block  on  the 
change  of  other  life  cycle  components  (i.e.  training,  security, 
etc.)  (*) 

Impact  of  discrepancy  fixes  on  other  life  cycle  components 
(i.e.  training,  security,  etc.)  (*) 

Updates  to  plans  for  the  implementation  of  life  cycle 
components  (*) 
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Table  6.1 

Specific 

Common 

Measurement 

Recommended 

Other  Candidate 

Issues 

Issue  Area 

Category 

Measure 

Measures 

Technology 

Technical 

Technology 

N/A 

Changes  in  the  technology  readiness  levels  (*) 

Readiness 

Adequacy 

Impacts 

Based  on  the  critical  success  criteria,  Table  6.2  identifies  the  rationale  for  selecting  a  specific  measure  for  a  specific  program. 
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Table  6.2 


Collection  Interval  with  Success 

Specific  Issues 

Candidate  Measures 

Rationale 

Guidelines 

1 .  Requirements  allocated  by 

Represents  tbe  work  to  be  done  for  a  block. 

Tracked  per  block  and  for  life  the 

block. 

The  numbers  of  actual  requirements  are 

of  program.  The  desire  is  to 

tracked  against  the  number  planned  for  a 

establish  an  average  number  of 

block.  Trends  in  the  number  of 

requirements  implemented  per 

requirements  allocated  to  blocks  are 
tracked  over  time. 

block. 

2.  Ratio  of  requirements 

Shows  growth  of  capability  of  the  system 

Long  term  tracking.  The  desire  is 

satisfied  in  a  block  to  total 

to  eventually  meet  full  system  capability. 

for  a  steadily  increasing  growth 

Flexible 

system  requirements. 

in  system  capability 

Requirements 

3.  Number  of  requirements 

Shows  the  volatility  of  requirements 

Tracked  per  block.  The  desire  is 

added,  deleted  &  modified 

allocated  to  a  block  causing  unplanned 

for  low  change  in  requirements 

per  block. 

rework  which  could  slip  schedule. 

per  block. 
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Table  6.2 


Specific  Issues 

Candidate  Measures 

Rationale 

Collection  Interval  with  Success 
Guidelines 

4.  Number  of  allocated  block 

This  measure  would  detect  slipping  new 

Tracked  per  block.  The  desire  is 

requirements  deferred  to 

system  capability  into  tbe  future.  Slipping 

for  low  slippage  of  requirements 

Flexible 

later  blocks. 

functionality  indicates  that  tbe  agreed-to 
block  requirements  were  unrealistic.  This 
possibly  means  the  allocation-decision 
process  may  not  be  effective.  Slippage  may 
also  mean  an  overall  increase  in  system  life 
cycle  cost. 

to  future  blocks. 

Requirements 

5.  Number  of  baseline 

This  measures  the  rework  necessary  on 

Tracked  per  block  and  for  life  of 

(cont.) 

requirements  changed. 

previously  implemented  requirements  that 
were  changed  due  to  deployment  of  blocks 
with  conflicting  interoperability  needs. 

This  measure  provides  the  basis  for 
planning  requirements  volatility. 

program.  The  desire  is  for  low 
change  in  the  requirements 
baseline. 

6.  Number  of  system 

The  system  requirements  are  the 

Tracked  for  the  life  of  program.  This 

requirements. 

foundation  from  which  each  block’s 
requirements  are  derived.  A  growth  or 
change  in  the  capability  of  the  system  will 
change  the  total  number  of  requirements  to 
be  implemented  and  the  ratio  of 
implemented  requirements  to  system 
requirements. 

measure  is  used  to  normalize  the 
ratio  and  trend  of  implemented 
requirements. 
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Table  6.2 


Specific  Issues 

Candidate  Measures 

Rationale 

Collection  Interval  with  Success 
Guidelines 

1 .  Risk  exposure 

Risk  exposure  (amount  of  risk  on  the 
project)  is  a  measure  to  determine  if  the 
Spiral  development  process  is  effectively 
reducing  the  risk  on  the  project. 

Tracked  long  term.  The  desire  is 
to  have  decreasing  exposure. 

2.  Number  of  risks  per  block 

This  measure  will  show  how  the  risks  on 
each  block  are  changing  over  time. 

Long  term  tracking.  The  desire  is 
to  decrease  the  numbers  of  risks. 

Risk 

Management 

3.  Number  of  risks  identified 
against  a  critical  technology 
component 

This  measure  is  critical  if  there  are 
significant  technology  risks  on  the  project. 

Tracked  per  block  with  high  risk 
components  dropped  from  the 
development. 

4.  Risk  carryover  from  previous 
block 

Carryover  is  a  way  of  measuring  the 
effectiveness  of  the  risk  mitigation 
strategies  that  are  planned  and  executed  in 
each  block  of  the  program. 

Tracked  per  block.  The  desire  is 
to  have  low  numbers  of  risks 
transfer  into  new  block 
developments. 

Technology 

Readiness 

1 .  Changes  in  the  technology 
readiness  levels  (section  4.3) 
for  system  components 

2.  Availability  of  one-of-a-kind 
test  beds  (T) 

Components  based  on  technology  that  is 
not  mature  can  delay  the  completion  of  a 
block.  Components  incorporated  in 
previous  blocks  may  continue  to  mature 
which  may  require  their  replacement. 

Unavailable  facilities  needed  for 
integration  and  testing  can  delay  the 
completion  of  a  block. 

Long  term  tracking  of 
components  across  blocks.  The 
desire  is  to  observe  increasing 
technology  readiness  of 
components 

Tracked  per  block.  The  desire  is 
to  have  as  close  to  100% 
availability  as  possible. 
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Table  6.2 


Specific  Issues 

Candidate  Measures 

Rationale 

Collection  Interval  with  Success 
Guidelines 

1 .  Documented  commitments 

These  two  critical  success  factors  are 

Tracked  per  block.  The  desired 

Involved 

by  relevant  stakeholders 

correlated,  i.e.  involved  stakeholders 
implies  a  trusting  culture. 

The  numbers  and  types  of  commitments  by 
stakeholders  are  used  for  block  planning. 

outcome  is  full  satisfaction  of 
commitments  by  stakeholder 
groups. 

Stakeholders 

2.  Documentation  and 

Measures  team  cooperation  and  interaction. 

Tracked  per  block.  The  desire  is 

and 

resolution  of  issues,  and 

to  have  input  from  all 

Trusting  Culture 

action  items  by  stakeholder. 

This  measure  shows  there  will  be  no 

stakeholder  groups. 

3.  Participation  by  stakeholders 

“surprises”  for  stakeholders  when  the  block 

Tracked  per  block.  The  desire  is 

in  demonstrations  or  testing 

is  released 

This  measure  shows  the  cumulative 

for  comments  or  change  requests 
to  be  submitted  by  all 
stakeholders. 

4.  Alignment  of  project 

satisfaction  of  stakeholder  needs  with  the 

Long  term  tracking  across  all 

performance  with  projected 
stakeholder  needs,  objectives, 
and  requirements. 

block  being  developed 

This  may  indicate  the  degree  of 

block  deployments.  The  desire  is 
an  increasing  list  of  satisfied 
stakeholder  requirements. 

5.  Requirements  changes  by 

incorporation  of  different  stakeholder 

Long  term  tracking.  The  desire  is 

stakeholder  type 

groups. 

that  no  one  group  is  excluded 
from  the  requirements  allocation 
decision  process  for  each  build. 
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Table  6.2 


Collection  Interval  with  Success 

Specific  Issues 

Candidate  Measures 

Rationale 

Guidelines 

1 .  Block  deployment 

This  measure  shows  the  number  of  planned 

Long  term  tracking.  The  desire  is 

milestones. 

blocks  and  the  duration  of  the  overall 

to  observe  on-time  deployment 

program. 

of  the  planned  blocks. 

2.  Inter-block  milestones  and 

This  measure  is  intended  to  show  the 

Long  term  tracking.  The  desire  is 

changes  in  block  schedule 

impact  of  inter-block  dependencies.  If 

to  observe  little  ripple  effect 

Schedule 

there  is  a  change  in  block  n  milestones, 

between  blocks  because  of 

how  does  this  impact  the  milestones  in 
future  blocks?  This  will  be  critical  if  there 
is  overlap  between  the  schedules  of 
multiple  blocks. 

milestone  slippage. 

3.  Build  Component  -  function 

Tracked  per  build.  The  desire  is 

This  measures  the  progress  of  functionality 

to  see  actual  meet  planned 

being  implemented  in  a  specific  build. 

implementation  of  functionality. 

4.  Test  case  status 

Tracked  per  build.  The  desire  is 

Measure  the  progress  of  successful 

to  see  progress  that  indicates  the 

completion  of  testing. 

completion  of  testing  to  make  the 
build  completion  date. 
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Table  6.2 


Specific  Issues 

Candidate  Measures 

Rationale 

Collection  Interval  with  Success 
Guidelines 

1 .  Percentage  of  unchanged  test 

This  measure  would  show  the  number  of 

Tracked  per  build.  The  desire  is 

System  Breakage 

cases  (from  previous  blocks) 

2.  Discrepancy  reports  against 

changes  to  regression  tests  as  a  result  of  the 
new  block.  It  is  an  indirect  measure  of  the 
stability  of  the  previously  delivered 
functionality 

to  see  little  change. 

previously  delivered 

This  would  measure  how  the  new 

Tracked  per  build  and  for  the 

functionality 

3.  Tracking  the  amount  of 

functionality  has  impacted  the  functionality 
(or  uncovered  problems  with  the 
functionality)  of  previous  blocks 

long  term.  The  desire  is  to  see 
minimal  breakage  in  previously 
delivered  functionality. 

rework  from  one  block  to  the 

This  is  a  measure  or  the  cost/effort  impact 

Tracked  per  build  and  for  the 

next 

4.  Discrepancy  reports  against 

of  system  breakage 

long  term.  The  desire  is  to  see 
minimal  rework  caused  by 
system  breakage. 

architecture 

This  is  a  measure  of  the  “robustness”  of  the 
original  architecture. 

Long  term  tracking.  The  desire  is 
to  observe  few  discrepancies 
concerning  the  architecture. 

19 


Evolutionary  Acquisition  Measures 


Table  6.2 


Specific  Issues 

Candidate  Measures 

Rationale 

Collection  Interval  with  Success 
Guidelines 

1 .  Impact  of  newly  allocated 

This  measure  will  show  the  impact  of 

Track  long  term  and  per  block. 

requirements  to  a  block  on 

implementing  additional  requirements  in  a 

The  desired  result  is  a  stable. 

Impact  to  Life 
Cycle  Factors 

the  change  of  other  life  cycle 
components  (impact  could 
be  cost  or  number  of  affected 
components) 

future  build 

average  life  cycle  cost  of 
implementing  new  requirements. 

2.  Impact  of  discrepancy  fixes 

This  measure  will  show  the  impact  of 

Track  long  term.  The  desired 

on  other  life  cycle 

fixing  a  discrepancy  in  one  block  on  the 

result  is  to  observe  a  minimal 

components  (i.e.  training, 

life  cycle  components  of  other  blocks  (both 

cost  to  the  rest  of  the  system  in 

security,  etc.) 

3.  Updates  to  plans  for  the 

delivered  and  future) 

As  each  block  is  defined  in  detail,  this 

fixing  descrepancies  within  a 
block. 

implementation  of  life  cycle 

would  measure  how  the  detailed  planning 

Track  long  term.  This 

components 

of  block  n  impacts  the  life  cycle  planning 
of  both  block  n,  and  all  other  (past  & 
future)  blocks 

information  can  be  used  in 
planning  the  amount  of  effort 
required  to  keep  plans  updated. 

20 


Evolutionary  Acquisition  Measures 


7.  Conclusion 

Evolutionary  acquisition  has  the  goal  of  reducing  cost  and  cycle  time  per  deployment  while  delivering 
improved  performance.  Cost  and  cycle  time  reduction  is  achieved  with  the  development  and  deployment 
of  small  blocks  of  functionality.  Improved  performance  is  achieved  hy  using  risk  reduction  strategies  and 
stakeholder  involvement  in  the  definition  and  building  of  the  blocks  of  functionality. 

Cost  and  cycle  time  are  traditionally  measured  and  reported  in  any  acquisition  strategy.  The  emphasis  on 
measuring  risk  and  stakeholder  involvement  provide  important,  new  insights  in  managing  an  evolutionary 
acquisition  project.  Important  challenges  to  the  evolutionary  acquisition  approach  are: 

•  avoiding  closed-ended  solutions 

•  trading  off  requirements  for  cost  and  schedule  in  a  block  delivery 

•  using  risk  to  set  development  priorities 

8.  Recommendations 

There  are  two  major  recommendations. 

1.  Select  an  organization  or  organizations  to  pilot  the  recommended  measures.  In  order  to  pilot  the 
measures,  it  will  be  necessary  that  the  organization  currently  is  doing  the  activity  that  is  being 
measured.  For  example,  in  order  to  pilot  the  risk  amount  measure,  the  organization  should  have  a 
defined  risk  analysis  process.  If  possible,  we  should  look  for  organizations  that  may  have  already 
used  a  similar  measure. 

2.  Develop  a  recommended  update  to  the  PSM  measurement  common  issues,  categories,  and 
measurement  specification  tables.  This  should  be  done  to  all  of  the  measures  that  appear  in  table  6.1. 
This  task  would  be  to  develop  a  recommended  update  the  version  4.0  PSM  common  issues, 
categories,  and  measurement  specification  tables,  and  present  this  update  to  the  PSM  organization  for 
potential  incorporation  into  the  current  PSM  materials. 
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Acronyms 

ARDEC 

CMM 

DOD 

ICM 

IOC 

KPP 

ORD 

OSD 

PSM 

SDM 

SEI 

S&T 

SPC 
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